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Abstract 

We consider optimal control of a stochastic network, where service is controlled to prevent 
buffer overflow. We use a risk-sensitive escape time criterion, which in comparison to the 
ordinary escape time criteria heavily penalizes exits which occur on short time intervals. A 
limit as the buffer sizes tend to infinity is considered. In [2] we showed that, for a large class 
of networks, the limit of the normalized cost agrees with the value function of a differential 
game. In this game, one player controls the service discipline (who to serve and whether to 
serve), and the other player chooses arrival and service rates in the network. The game's value 
is characterized in [2] as the unique solution to a Hamilton- Jacobi-Bellman Partial Differential 
Equation (PDE). In the current paper we apply this general theory to the important case of a 
network of queues in tandem. Our main results are: (i) the construction of an explicit solution 
to the corresponding PDE, and (ii) drawing out the implications for optimal risk-sensitive and 
robust regulation of the network. In particular, the following general principle can be extracted. 
To avoid buffer overflow there is a natural competition between two tendencies. One may choose 
to serve a particular queue, since that will help prevent its own buffer from overflowing, or one 
may prefer to stop service, with the goal of preventing overflow of buffers further down the line. 
The solution to the PDE indicates the optimal choice between these two, specifying the parts of 
the state space where each queue must be served (so as not to lose optimality), and where it can 
idle. Referring to those queues which must be served as bottlenecks, one can use the solution 
to the PDE to explicitly calculate the bottleneck queues as a function of the system's state, in 
terms of a simple set of equations. 



1991 Mathematics Subject Classification. Primary 60F10, 60K25; Secondary 49N70, 93E20. 

*This research was supported in part by the United States-Israel Binational Science Foundation (BSF 1999179) 
^Department of Electrical Engineering, Technion - Israel Institute of Technology, Haifa 32000, Israel. 

atar@ee.technion.ac.il. Research of this author also supported in part by the fund for promotion of research 

at the Technion. 

*Lefschetz Center for Dynamical Systems, Division of Applied Mathematics, Brown University, Providence, R.I. 
02912. dupuis@dam.brown.edu. Research of this author also supported in part by the National Science Foundation 
(NSF-DMS-0072004, NSF-ECS-9979250) and the Army Research Office (DAAD19-02- 1-0425). 

^Department of Electrical Engineering, Technion - Israel Institute of Technology, Haifa 32000, Israel. 
adam@ee.technion.ac.il. Research of this author also supported in part by INTAS grant 265, and in part by 
the fund for promotion of research at the Technion. 



1 



1 Introduction 



In a previous work we considered a stochastic control problem for a Markovian queueing network 
with deterministic routing, where the service stations may provide service to one or more queues, 
with each queue being limited by a finite buffer. The control refers to the service discipline at 
the service stations, and the cost involves the time till one of the queues first reaches its buffer 
limit. Such a problem can be regarded as the control of a Markov process up to the time it exits 
a domain, the domain being the rectangle associated with the buffer sizes. The cost is chosen so 
as to obtain a risk-sensitive (or rare-event) control problem: one considers E x e~ ca as a criterion 
to be minimized, where c > is fixed and a denotes the exit time (the time when any one of 
the buffers first overflows). Such a criterion penalizes short exit times more heavily than ordinary 
escape time criteria (such as E x a, a criterion to be maximized). While the main result of .2 is the 
characterization of a limiting problem as the buffer sizes tend to infinity, the current work focuses 
on finding explicit solutions to this limit problem and on the interpretation thereof. 

There are at least two motivations for the use of risk-sensitive criteria when designing policies 
for the control of a network. The first is that in many communication networks performance 
is measured in terms of the occurrence of rare but critical events. Buffer overflow is a principal 
example of such an event. The second motivation follows from the connection between risk-sensitive 
controls and robust controls. Indeed, as discussed in jS], the optimization of a single fixed stochastic 
network with respect to a risk-sensitive cost criteria automatically produces controls with specific 
and predictable robust properties. In particular, these controls give good performance for a family 
of perturbed network models (where the perturbation is around the design model and the size of 
the perturbation is measured by relative entropy), and with respect to a corresponding ordinary 
(i.e., not risk-sensitive) cost. 

In many problems, one considers the limit of the risk-sensitive problem as a scaling parameter 
of the system converges, in the hope that the limit model is more tractable. This is the path 
followed in in the asymptotic regime where time is accelerated and buffer lengths are enlarged 
by a factor n. The limit of the normalized cost was characterized both as the value function of a 
differential game, and as the solution to a corresponding nonlinear PDE. Both interpretations are 
important and useful. It is the interpretation as the value function of a differential game that is 
key in quantifying the robust aspects of the resulting control policy. However, because the PDE 
gives a necessary and sufficient characterization of the (a priori unknown) value function, it can 
provide qualitative information on the structure of the value function and the optimal controls. In 
particularly favorable circumstances one can go even further, and use the PDE for exact calculation 
and control policy synthesis. Indeed, if one can by any means guess a proper parametric form of 
the value function, then it is sometimes possible to use the PDE to verify that this is the correct 
form, and identify the unknown parameters in the representation for the value function. The 
instances in the control theory literature where this has been carried out are few and far between, 
especially when the state space of the system has dimension greater than two. However, important 
information is obtained from these instances on the structure of optimal controls. 

In this paper the focus is on applying the PDE obtained in 2 in the manner just described. 
In particular, we treat in detail the general case of a tandem queueing network, and construct an 
explicit solution to the PDE. In a system of queues in tandem, each server offers service to exactly 



2 



one queue, and therefore the service control refers simply to whether each station should provide 
service or remain idle. Naturally, it is important to provide service to a queue in order to keep it 
from reaching the buffer limit. On the other hand, if the next buffer in line is nearly full, it might 
be necessary to idle the first queue in order to keep the second from overflowing. Besides factoring 
in how close all buffers are to their respective limits, one must consider the mean service rates and 
the likelihood of significant deviations from those mean service rates. Is it likely that the next 
queue down will stall and simply stop serving for a while? It is also possible that one will have to 
look even further ahead, and consider buffers further downstream. 

Although the optimal control problem for the Markovian queueing system is fully described 
by a dynamic programming equation [2j, such equations are typically solved numerically. It is 
hard to extract any global structural information from the exact equation, and even a numerical 
solution may not be feasible when buffers are large or the dimension of the state space is moderate. 
As discussed previously, the solution to the PDE for the limiting problem turns out to simplify 
things significantly. Roughly speaking, the PDE indicates the following structure of the asymptotic 
optimally controlled network. In the interior of the domain (i.e., when all buffers are away from 
their maximum capacity), and depending on the state of the system, service must be provided at 
certain service stations, while other stations may idle without causing loss of optimality. In that 
sense, the limit problem sharpens the control policy by emphasizing the importance of serving 
those 'bottleneck' buffers (it is crucial to serve the bottleneck buffers, and completely unimportant 
to serve the others). The identification of the bottlenecks is nontrivial, and indeed queues with 
the smallest service rate are not necessarily bottlenecks. Instead, as hinted above, identification of 
the bottlenecks must include consideration of at least the following: (i) the relative closeness of all 
buffers to their maximum value, (ii) relative mean service rates, and (iii) relative uncertainties in the 
service rates. We identify a system of algebraic equations whose solution identifies all bottlenecks. 

There is relatively little work on risk-sensitive and robust control of networks. Ball et. al. have 
considered a robust formulation for network problems arising in vehicular traffic 0, where the 
cost structure is qualitatively different. Dupuis studies a robust control problem for networks in a 
deterministic setting and obtains explicit solutions for the value function 0. The cost there is, in a 
sense, antipodal to the one considered in the current work, namely the time till the system becomes 
empty (a criterion to be minimized). For other recent work on queueing control in a large deviation 
regime see Stolyar and Ramanan JH], where a single server non-Mar kovian system (with quite 
general stationary increments input flows) is studied, and a particularly simple scheduling control 
policy is shown to be asymptotically optimal (see also an extension of the work in Stolyar |14j). 

The paper is organized as follows. Section 2 introduces the model and the PDE and states 
the main result. Section 3 contains discussion and interpretation. In Section 4 we prove the main 
result. 

Notation: The symbol V stands for maximum, while A stands for minimum. Denote scalar 
product between two vectors as x ■ y. For integers i < j, let = {!,... ,j}. 
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2 Model and preliminaries 



The queueing network control problem. We consider the following tandem network. There 
are J queues and J servers. Customers present at queue i at a certain time are said to be of class i 
at that time. Customers arrive to queue 1 according to a Poisson process of rate A > 0. Service at 
queue i is provided by server i at exponential time with parameter fii > 0, mutually independent 
and independent of the arrivals. After a customer is served by server i, it moves to queue r(i), 
where r{i) = i + 1, i = 1, . . . , J — 1, r(J) = 0, and i = is used to denote the "outside". The state 
of the network is the vector of queue lengths, denoted by X. Let {ef, i = 1, . . . , J} denote the unit 
coordinate vectors, let eo = and denote 

7i = e . -e r(i) . (1) 

Note that 7« = — e^+i for i = 1, . . . , J — 1, jj = ej, and that following service to queue i the state 
changes by — 7 j. The control is specified by the vector u = (u±, ...,uj), where ui = 1 if customers in 
queue i are given service and Uj = otherwise. We next consider the scaled process X n under the 
scaling which accelerates time by a factor of n and down-scales space by the same factor. We are 
interested in a risk-sensitive cost functional that is associated with exit from a bounded set. Let G 
be the rectangle defined through 

G = {(xi, . . . ,xj) : < xi < zi; < Xi < Zi, i = 2, . . . , J}, (2) 

for some > 0, i = 1, . . . , J. Note that G contains parts of, but not all of its boundary. Let 

a n = m£{t : X n (t) G}. 

The control problem is to minimize the cost E x e~ nccjn , where E x denotes expectation starting from 
x, and c > is a constant. With this cost structure "risk-sensitivity" means that atypically short 
exit times are weighted heavily by the cost. As a result, even if exit within short time occurs 
with small probability, it may have a noticeable effect on the cost, and thus a "good" control will 
attempt to avoid such events as much as possible. 

Although this paper do not directly treat the stochastic control problem but the corresponding 
PDE, we give here the precise formulation of the former, for completeness. For each n, the state 
space is G n = n~ l, L J + fl G. The control (or action) space is given by 

U = {( Ui ),i = l,...,J :0< Ui < 1, i = l,...,J}. 

One considers state processes X n that are similar to those defined for the original queueing network, 
except that time is accelerated (equivalently, arrival and service rates are multiplied) by a scaling 
parameter n, and space is scaled down by the same factor. More precisely, for u G U and / : — > 
R, let 

J 

C u f(x) = X[f(x + ei ) - f(x)] + «i/*l{x-7 ie z£}[/(* - 7i) " /(*)], x e Z J + . 

i=i 

For n € N let 

jC n ' u f(x) = n£ u g(nx), 
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Figure 1: A simple queueing network, a rectangular domain and three types of boundary. Full 
line: d+G, dashed line: d c G, and dotted line: d G 

where / : n~ lr L + — > M and g(-) = f(n~ 1 -). A controlled Markov process starting from x G G n will 
consist of a complete filtered probability space (fl,^ 7 , (Ft), Px' u ), a state process X n taking values 
in G n and a control process u taking values in U, such that X n is adapted to J-~t, u is measurable 
and adapted to JF t , P"' n (X n (0) = x) = 1, and for every function / : G n — > M 

f(X n (t))- f C n ' u ^f{X n {s))ds 
Jo 

is an .Fj-martingale. £'™' lt denotes expectation with respect to P"' u . For a parameter c > 0, the 
value function for the stochastic control problem is defined by 

V n {x) = -\nin- 1 log E^ n e- nccTn , x G G n , (3) 

where the infimum is over all controlled Markov processes. 

The domain and its boundary. It is possible for the controller to prevent any but the first queue 
from exceeding Zi, simply by turning off service to the preceding queue. However, the controller 
cannot prevent overflow of the first queue. Although it is in principle possible that the dynamics 
could exit through the portion of the boundary defined by queues 2, . . . , J, it is always optimal for 
the controller to not allow this. Consider the simple two-class network illustrated in Figure ^ The 
controller can prevent exit through the dashed portion of the boundary simply by stopping service 
at the first queue. As a consequence, there are in general three different types of boundary-the 
constraining boundary due to non-negativity constraints on queue length, the part of the boundary 
where exit can be blocked, and the remainder. These three types of boundary behavior result in 
the PDE in three types of boundary conditions. The three portions of the boundary are explicitly 
given as 

d c G = {(zi, ■ ■ ■ , Xj) : < x\ < zi, and xi = z^ for somei > 1} 

d Q G = {(xi, . . . , xj) : x\ = zi, and < X{ < Z{ for all i > 1} 

d + G = {(xi, . . . , xj) : Xi < Zi for all i, and xi = for some i}. 

Note that d c G, d Q G and d+G partition the boundary dG of G. Also, d c G C G while d Q G n G = 0. 
As usual, we will denote G° = G \ dG and G = G U dG. 

The Hamiltonian, PDE and viscosity solutions. It is a standard fact that the value functions 
V n of the stochastic control problem considered above can be characterized by a dynamic program- 
ming equation. The results of [2] show that also the limit of V n as n — > oo can be characterized by 



a certain equation, namely a Hamilton-Jacobi-Bellman PDE, that can in fact be regarded as the 
dynamic programming equation for a certain deterministic game problem. For details on this PDE 
we need some notation (the reader is referred to [2] for details on the game). Let I : R — > M+U{+oo} 
be defined as 



l(x) 



x log x — X + 1 £ > 0, 

+00 x < 0, 



where OlogO = 0. Define 

M = {m = (A, /2i, . . . ,/2j) : A > 0, fa > 0}. 
For u € U and m E M define 



v{u,m) = Aei — 



i=l 



i=l 



p(u, m) = XI ( y ) + ^ Uifiil f — 



/i, 



where 7$ are as in Let 

it, m) = c + p ■ v(u, m) + p(u, m) 



c + 



Xpi + XI 



i=l 



-pm - p + mi 



(4) 



(5) 



and let the Hamiltonian be defined as 



H{p) = sup inf H(p,u,m) = inf sup H(p, u, m), (6) 

where the last identity, expressing the Isaacs Condition is proved in [2] . The following simplification 
in the structure of the Hamiltonian will be useful. Using convexity and the fact that the slope of 
I at + is —00, the minimum over m is attained at A = Xe~ Pl , fii = pie li ' p . A straightforward 
calculation then shows that 

H(p, u) = inf H{p, u,m) = c + [(A — A) + Ui(m - fU)] 

m 

J 

= c + A(l - e" Pl ) + - e^ p ). (7) 

i=l 

Define I(x) = {i : Xj = 0}. The PDE of interest is the following. 

' H(DV(x)) = 0, i£ G°, 

< DV(x) • 7i = 0, i £ I(x), x G a+G, (8) 
, V(x) = 0, i£ 9 G. 
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Since typically such equations do not possess classical solutions, the framework of viscosity solutions 
is useful (see (Hj). This framework allows for functions that are merely continuous to be regarded as 
solutions. For x G G, the set of superdifferentials D + V(x) and the set of subdifferentials D~V(x) 
are defined as 

D+V(x) = L : hmsup ^-^-f- {lJ ~ X) < ol , (9) 
I v^x \y-x\ J 

d-v(x) = L : lM v (y)-v(*)-P-(y-x) > J (10) 
I y-* x \y — x\ J 

Definition 1 V is a viscosity solution to if 

H(p) V max p ■ j t > 0, p G D + V(x), x G G, (11) 

ie/(a;) 

ff(p) A min p • 7i < 0, p G x £ G \ d c G, (12) 

i£/(a;) 

and V(s) = for x G c? G. 

The following is a special case of Theorem 2 of 

Theorem 1 There exists a unique Lipschitz viscosity solution V to Moreover, if x n G G n , 
n G N are suc/i that x n — ► x G G, t/ien lim n ^oo y n (x n ) = V(x). 

For i = 1, . . . , J, let /3j denote the unique positive solution to 

c + A(l-e ft ) + ^(l-e" ft ) = 0. (13) 
Set 6j = (5i Y?j=i e j- Our main result is the following. 

Theorem 2 Assume c > 0. T/ien the viscosity solution to the PDE is given by 

V{x) = min bi ■ (z — x). (14) 
i=l,..., J 

3 Some remarks 
3.1 Interpretation 

As commented above, it is possible to characterize the value function V n as the solution to a 
dynamic programming equation on the discrete space G n . Moreover, if V n is available in explicit 
form, one can use the dynamic programming equation to specify an optimal control policy for the 
problem. The results above provide an explicit expression only to the limit V = lim n V n , and 
therefore we are unable to specify an optimal policy for the problem with finite n. Instead, we shall 
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use the quantity V along with the HJB equation (JSJ) to propose a policy for the queueing network 
problem (and the corresponding value V n ). It is plausible that the proposed policy is, in a sense, 
asymptotically optimal as n — > oo but we do not attempt to prove such a statement here. Instead, 
the discussion below could be regarded as a natural interpretation of the result. Moreover, since V 
can be considered as a good approximation to V n (for large n), it can be used as an initial condition 
in a value iteration procedure that calculates V n and an optimal policy. 

The solution to the PDE stated here may be interpreted as follows. Let x be an interior point 
of G, and assume that V is differentiable at x. The form (|14|) implies that for some j = j(x), 

3 

DV(x) = -b j = -/3 j J2 e i- ( 15 ) 



i=i 



Hence by ©, the equation H(—bj) = has the form 



sup 



c + A(l - + UjHj(l - e'to) 



0. 



It is seen that the supremum is attained at any u for which Uj = 1 and G [0,1], i / j. The 
interpretation in terms of the service policy is that it is important to serve class j, while optimality 
does not depend on the service given to other classes. The station j = j(x) can therefore be regarded 
as a bottleneck: it is crucial to serve station j when at state x. Since the PDE describes the limit 
of the stochastic control problem, one expects the bottleneck stations to have a similar property in 
the stochastic problem: Although it may be optimal for all servers to not idle when the system is 
near an interior point x, it is significant for the bottleneck server to work while if the other servers 
idle the cost is affected only by little. 

A closer look at ()15|) reveals that all bottleneck stations belong to a certain set A' defined below. 
More precisely, given an interior point x where V is differentiable and denoting y = z — x, using 
(|14|) , we see that (I15j) holds for j if and only if j satisfies 

Pj(yi + --- + Vj)<0i(vi + -~ + Vi) V^i. (16) 

We claim that a necessary condition for j to satisfy Ijlfijl (for some y) is j G A' , where 

A' = {k G [1, J] : /U fe < m, for all I < k}, 
and by convention, 1 G A' . To this end, write the explicit form of the solution to (|13j) as 

e ft = (2A)- 1 [c + A + W + (c 2 + A 2 + /^ 2 + 2c(A + W )) 1 / 2 ], 
and note that the positive solutions /3j to (|13j) are monotone in /Xj, in the sense that 

Hi < fij Pi < Pj. (17) 

If (US!) holds then < for j = 1, . . . , i, and it follows from (JT7J) that j G A'. 

It should be emphasized that the condition j G A' is only necessary for j to be a bottleneck, and 
being a bottleneck is really a function of the system's state. A more complicated necessary condition 
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is given in Section 0J which is of the form j € A{x). One interesting and perhaps counterintuitive 
phenomenon is that it is possible that a station is nearly empty while others are far from being 
empty, and still it is a bottleneck. 

In the rest of this subsection we discuss a different queueing system for which a similar analysis 
is possible, including an explicit form for the limit of the value function and a clear interpretation 
of it regarding nearly optimal service policies for the queueing system (see [2] for proof of the 
results quoted below). Consider a single server that provides service to J classes (each customer 
requires service once) . Service rate to queue i is exponential with rate \i{ , and the arrival process to 
class % is Poisson of parameter A, > 0, % = 1, . . . , J. All arrival and service processes are mutually 
independent. One considers the control space 

U = / u : ^2 < > 0,z = 1,..., j\ , 

where Ui represents the fraction of service allocated to class i by the single server. Analogously to 
the problem discussed above, one defines a domain G = {(xi, . . . , xj) : < Xi < Zi, 1 = 1, . . . , J}, 
scaled queueing processes X n and exit time a n , and attempts minimizing the cost E x e~ nayn over 
an appropriate class of control processes. Let V n {x) denote the infimum. One can then show that 
lim n V n (x n ) exits whenever x n — > x € G. Moreover, the limit V is characterized in terms of a PDE 
of the form (JHJ), where now 7j = e$, i = 1, . . . , J, and H (p) = sup ugC/ H(p, u). Here H (p, u) has the 
form (analogous to ®): 

J 

H(p,u) = c + J2[W-e- pi ) + Ui^(l-eP% (18) 

i=\ 

and the limit V{x) can be explicitly calculated provided c is large enough as follows. For i= 1, . . . , J, 
define as the unique positive solution to c + Aj(l — e ai ) + — e~ ai ) = 0, namely 

e a * = (2A)- 1 [ c + X + fH + ((c + \ + ^) 2 - 4A/x 4 ) 1/2 ] . 

Then if c is large enough one has 

V(x) = min aj(zj — Xj). (19) 

i=l,...,J 

Let x be a point in the interior of G where the gradient of V is well defined. Then the PDE is 
satisfied in the classical sense at this point. Thus, H(DV{x)) = H(-ajej) = 0, where j = j(x). 
Using pH|>. the equation H(-aiej) = takes the form 

sup [c + Aj(l - e a + UjHj(l - e~ a 0] = 0. 

Clearly the supremum is attained at U{ = li=j, i = 1, • • ■ , J- This means that it is optimal to serve 
class j{x) at the state x. In the totally symmetric case, where ^ = [A, Aj = A, Zi = z for all i, the 
solution takes the form V(x) = a minj(z — Xj), and the optimal service discipline can be interpreted 
as "serve the longest queue." An asymmetric two dimensional example is given in Figure [21 where 
the domain G is divided into two subdomains G% and G*2 in accordance with the structure (|19j) . 
and the optimal service discipline corresponds to giving priority to class i when the state is within 
Gi, i = 1, 2. This discipline gives priority to the queue with the (weighted) shortest free buffer space. 
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Figure 2: Priority to class % when the state is in Gj, i = 1, 2. 



3.2 The perturbed rates 

In the asymptotic analysis of rare events it is often the case that most of the probability mass of an 
event of interest is concentrated on the event that the stochastic process, say X n , nearly follows a 
certain deterministic path, for large values of n. For example, the most likely way a stable M/M/l 
queue overflows is by nearly following a deterministic path that is the solution to a fluid model 
in which arrival and service rates are reversed. This appears as a consequence of time-reversal 
arguments in [12]. For many other cases where one can compute deterministic paths about which 
most mass is concentrated see )13j . Similarly, in the current stochastic control problem, when 
the system operates under the optimal control most contribution to the cost is obtained when the 
process X n nearly follows a certain deterministic path. This path is now the solution to a fluid 
model with a control u and perturbed arrival and service rates m = (A,/2i, . . . , p,j). At a point x 
where the viscosity solution V to (JBJ is differentiable, the correct values of u and m are those that 
achieve the max-min in H(DV(x)) = s\xp u ini m H(DV(x),u,m) (cf. ©). 

We remark that in the example of tandem queues the optimal perturbed rates satisfy certain 
relations with the unperturbed rates, as shown in the two equations below. By ([7|l. the relation 
H(DV) = implies 

J J 
X + ^Uifii = c + X + ^Uifii . (20) 
i=l i=l 

Moreover, by l(7|). by the fact that the minimum over m is attained at A = Xe~ pi , p>i = /ije 7i ' p , and 
using Yli=i 7i = e i> we have 

J J 

xYlfii = X]liH. (21) 

i=l i=l 

Relation (|20|) was noticed by Avram The relation 1)21)1 appears to be new. A generalization of 
these equations to a more general network is possible and will appear elsewhere. 

Remark: An interesting relation between the roots q.% and (5i and the busy cycle period was pointed 
out to us by Boxma [If. For example, if B denotes the busy cycle period for an M/M/l queue 
under the stationary distribution, then Ee cB = e 13 , where 

c + A(l-e /3 ) + ^(l-e- /3 ) = 

(compare with 1)13)1 ) . 
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4 Proof of the main result 



Recall that I(x) = {i £ [1, J] : X{ = 0}, and denote B(x) = {i G [1, J] : Xj = Note that J(x) 
and i?(x) do not intersect. Since points x for which x\ = z\ are not in G, we have 1 G" £>(x) for all 
xeG. 

We show that the minimum over i = 1, . . . , J in (|14|) can equivalently be performed over a 
smaller (state-dependent) set. As in Section |31 let 

A' = {ie [1, J] : m < Hj, for all j < i}, (22) 

and by convention let 1 G A'. Let also 

A(x) = {i 6 i' : either i + 1 G" B(x); oi j > i and + 1, j] C B(x) imply < Hj}- 

Lemma 1 The minimum in ()14|) is obtained over the indices in the set A{x) defined above. More 
precisely, 

V(x) = min bi ■ (z — x) = min bi ■ (z — x) . 
i=l,...,J i£A(x) 

Proof: Fix i£G and recall that z\ — x\ > 0. If 

bi ■ (z — x) < minftj ■ (z — x), (23) 

then Pi < Pj for j G and it follows from JT2J) that i G A'. Thus 

y(x) = min bi ■ (z — x) = min&j ■ (z — x) 

i=l,...,J i£A' 

for all x G G (although not every i G A' is a minimizer). Fix now i Gi', and suppose that 

there is some j > i so that [i + 1, j] C -B(x) and [i% — Mi- (24) 

Since z G A', if i + 1 j4' then //j+i > /Zj. But then, if in addition % + 2 j4' we have /ii+2 > Mi- 
So, ii k A' for all i + 1 < k < j — 1 then fj,k > Mi) an d together with /ij < [ii we have that j & A' . 
We conclude that under (|2*4"|) . 

there is A; G [i + 1, j] such that k G ^4'. (25) 

By (EH), A > Pj and so 

6i- (z-aj) > bj ■ (z-x). (26) 
Therefore, for k as in (|25|) and under the assumptions in ()24j) . 

6i • (2 - x) > b k ■ (z - x), 

and since k G A' , we need not consider i in the minimum. As a result, ()14j) is equivalently given as 

V(x) = min bi ■ (z — x), x G G, 

i&A(x) 
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where A(x) is obtained from A'(x) by deleting those i G A'{x) which are followed by empty queues, 
[i + C B(x), and such that /ij > fij. This leaves in A{x) an index i from A'(x) only if either 
Zi + \ > Xi + i, or Zk = Xk for i + 1 < k < j implies fii < fij. More formally, 

A(x) = A' \ {i G A' : 3 j > i, [i + 1, j] C W > /Xj} 

= {i G ^4 : either z + 1 ^ B(x); or j > i and [i + 1, j] C B(x) imply ^ < fij}. (27) 



The form of the proposed solution is the minimum of smooth functions. The set of superdiffer- 
entials at a point where the function is not smooth can be seen (using the definition Q) to consist 
of the convex hull of the gradients of the smooth functions defining it, at that point. Also, at the 
boundary, the fact that there are less constraints introduced by (jUJ) on p G D + V than there are 
when x is in the interior, has an effect of enlarging the set further. Thus, for example, the set 
of superdifferentials of the zero function from M + to M at zero is M+. Using these considerations 
and the rectangular structure of the domain, one finds the general form for the superdifferential as 
follows. Let x G G be fixed. Set A = A(x), I = I(x), B = B(x), and O = {1, . . . , J} \ (J U B). 
Then any element p G D + V{x) is given as 

P = ~ X, Vihi + ^ 
ieA 

for some where 

Si > 0, i G I, Si < 0, i G B, Si = 0, t'G O, (28) 

and 

Vi > 0, i G A, Vi = 0, i g" A, ^2^ = 1. (29) 

If we denote 

Sj+t = 0, (30) 
then, using Q, the Hamiltonian is expressed as 

J 

ff(p)=c + A(l-e- pi ) + ^OV W (l-e") 

i=l 

J 

= c + A(l - "ifo-h) +J]0Vft(l- g-^A+^-^+i). ( 31 ) 

i=i 

Proof of Theorem |2J 

Verifying the PDE for superdifferentials. To verify it suffices to show that i?(p) > whenever 
p G I?y + (x) and p ■ ji < for all i E I. 

Step 1. We show first that -fT(p) > whenever p G DV + (x) and p • 7i < for all i = 1, . . . , J. The 
forms of bi and 7« = — e^+i imply the last inequality can be rewritten 

-Vifo + Si- S i+ i < 0, i = 1, . . . , J. (32) 
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In this case, (|3*Tj) becomes 

J 

i=i 

The constraints we have put on (u, 5) define a convex set S, namely, the set of (u, 5) € M. J x M J+1 
satisfying (|2"5|) . lf2"§|) . (|3*U)) and (|3~2"|) . The set S is bounded for the following reasons. First, since v 
is a probability vector (see (|HJ)) we have i/j £ [0, 1]. Next, by (|2"8)) and 1 £?, <5i > 0. Finally, it 
follows from (|3*2"|) (using the convention Y? s = if i < s) that 

i-l J 

5t - ^2 ^ 6 i ^ X] « = i , • • • , j r . 

This shows the boundedness. 

The function (y, 5) *— > /t(z/, 5) is concave, and therefore to prove /i > on S it suffices to check 
the inequality on the set E of extreme points of S, which contains a finite number of points since 
the constraints are linear. 

Lemma 2 All points (v, 5) £ E have the following form: 

Vf. = l for some k, and 
s (r,k) = p k (i.^ i+1 _ i = 1, . . . , J, r = s, . . . , t - 1, 

where t, s are defined below. 

Proof: We will first obtain the general form of v and then that of 5. 

We claim that for any (y, 5) £ E, v is of the form v = 1%, (short for Vi = li=k, i = 1, . . . , J), 
for some k £ A. Assume this is false. Then there are l,m 6 A, I < m, for which vi,v m £ (0,1). 
We will show that there is a vector A m , such that replacing {y, 8) by (z/, 5') = {y — ee m , <5 + eA m ) 
maintains the relations Q28|). (j30J) and Q32|) for both e > and e < (provided |e| is small). A 
similar statement will hold also for (y" , 5") = (y + eej — ee m , 5 — eA; + eA m ), and as a result all 
of (JUJ), (|3UJ) and (|311) will hold for both e > and e < 0, a contradiction to (u,5) being an 

extreme point of S. The construction of A m based on (y, 5) can be mimicked to construct A; based 
on (y 1 ', 5') (in particular, no use is made of the fact that V{ sum to one, but only that some of its 
components are within (0, 1)), and therefore the latter construction is omitted. 

Case 1: Inequality (|32|) holds as a strict inequality for i = m, i.e., —VmPm + o~m ~ 3m+i 

< 0. 

Then — (y m — e)(3 m + 5 m — 5 m +i < also holds (for both e positive and negative), provided that |e| 
is small. Here we take A m = 0. We see that {y — ee m , 5) satisfies the requirements. 

Case 2: Inequality §32$ holds with equality for i = m, i.e., —Vmfim + 5m — <Wfl = 0. Since 
VmPm > 0, either 5 m > or S m+ i < 0. Assume S m > (the case 5 m +i < can be treated 
analogously, and is therefore omitted). Let m' be the smallest j € [l,m] for which ()32|) holds with 
equality for all i £ [j, m\. For i £ [m',m — 1] (the set being empty and the statement void if 
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m' = m) we have — + 8i — 8i + \ = 0, hence if 8i + \ > then 8{ > 0. Together with the fact that 
8 m > 0, this shows that 

5i > 0, i e [m',m]. (34) 

Moreover, 

—Vm'-lPm'-\ + <W-i — 5 m ' < (35) 

(the statement being void in case that m! = 1). Set A m = -/3 m ^]™ m , e;, v' = v + ee m , and 
(5' = (5 + eA m . The perturbation eA m is chosen to cancel the change in —v m (3 m , and to preserve the 
left hand side of 1)32)1 for alH > m' — l. However, by (|35j) the inequality is maintained for i = m' — 1 
as well, by taking |e| sufficiently small. Similarly, by 1)341) . I[28[) and ()3U|) are also maintained on 
taking [e| small. 

This completes the construction of A m . As described above, this leads to a contradiction, and 
we conclude that any extreme point (y, S) S E satisfies v = 1^ for some k € A. 



When v = If. for some k G A, inequalities (|32|) can be rewritten as 

<5i < Si+i, i^k, 5 k < Pk + Sk+i- (1321) 

In particular, < 8\ < ■ ■ • < 8^. Let s denote the largest j < k for which j S BUO (and s = if 
there is no such j). The definitions of O and B imply 5 S = 0, and thus 

<5j = for 1 < j < s. (36a) 

Similarly, 0£;+i < • • • < 8j < 0, and if t denotes the least j G [k + 1, J] for which j E / U O (and 
t = J + 1 if empty) , then 

8j =0, t<j<J. (366) 



Hence (5 must satisfy 




In addition, 




< $k < 0k + S k+ i, whens</c-l, 

(36c) 

when s = k. 



< St-i < 0, when t>k + 2 
when t = k + 1. 



We have just shown that (|28j). (|3T?|) and (|32l ) imply (36a-(i). On the other hand, clearly (36a-d) 
implies 1)321). Moreover, as follows directly from the definition of s and i, 

i G / for all i E [s + l,k] and i E B for all i G [k + 1, i - 1]. (37) 

This shows that (36a-d) implies |(2E|) and J3DJ). Thus J2HI), © and (jS2J) are equivalent to (36a-d). 
Now, the set of 8 satisfying the constraints (36a-d) is easy to analyze. In particular, it is not hard 
to see that it has the following t — s extreme points, indexed by k and r G [s, . . . ,t — 1], namely 

£- r,fc) = /3fc(lj> r+ i - lj>fc + i), i = 1,. .. , J, r = s, ... ,t - 1. (38) 
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We now calculate h (cf. (|33|) ) at each extreme point. To this end, note that if (u, 5) = (lk,S^ r ' k ^) 
then by (35) 

I — fik i = r, 
-Vifc +5i- 5 i+ i = < i = 1, . . . , J, 

[0 i^r, 

and 

(A r = 0, 
[0 r > 0. 

Substituting in (|33]> . we have the following possibilities. If r = then = /3/-l ie r 1)fe i thus 

X] — 5i = 0, and —Vii3% + 5i — 5i + \ = 0, i = 1, . . . , J. Hence h(lk, 5^'^) = c > 0. Otherwise, 

h(l k ,5<- r <Q) = c+A(l-e & ) + Mr(l-e- /3fe ). (39) 

In case that r = k, the right hand side of (j39|) vanishes owing to the definition of /3& (see (|13|) ). 
In case that r < k, recall that k G A, and in particular, k G j4'. By (|22|) we therefore have 
Mfc — Mr> hence by (fl3|) . /i(lfc, <j( r ' fe J) > 0. Finally, consider the case where k < r < t — 1. By (|37|) . 
[k + 1, r] C B. Since k G A, (|27|) implies that /i^ < fi r , and we again conclude that h(l k , 5^) > 0. 
Having shown that h(is, S) > for all extreme points of the set S 1 , we conclude that the inequality 
holds for all (v, 5) G S. 

Step 2. We now relax the condition (|3*2"|) . Let then (^, <5) satisfy (|2"%|) . (|2ll|) . (|3*n|) . and assume that 
the inequality in (|32|) holds for all z G J (relaxing the assumption made in Step 1, that it holds for 
all 1 < % < J). Let P = P(f , <5) denote the set of i such that 

-z/iA + 5i- 5 i+1 > 0. (40) 

If P is empty then the results of Step 1 apply and H(p) > 0. Hence assume P is not empty. Let 
j = j(u, 5) be the least element in P. 

Note that J G" P. For if J G P then using ((3DJ) and (jSOJ we find (5j > 0, and therefore J G I. 
However, we get to assume (j32|) for all i E I, which means that —vj(3j + Sj — <5j+i < 0. This 
contradicts J £ P. 

We proceed by backward induction on the value of j(u, 6). Note that the sets B and I depend 
on x. The argument below treats simultaneously all x G G by considering all possible sets B and 
I. 

Induction step. Assumption: For all (I, B, v, 8) satisfying J2H), ((221), ©, such that P(i/, <5) D J = 
and 5) G [i + 1, i + 2, . . . , J - 1], one has ff(p) > 0. Let (I, B, u, 5) be such that P(y, 5) f)I = 
and j(v,S) = i (i > 1). Then — + <5j — Jj+i > 0. Modify <5j + i by increasing it so as to get 
equality i.e., set 5' i+1 > 5{ + i so that — Vify + <5j — = 0. This modification does not change the 
value of V //j(l — e _I/i ^ + ' 54 ~' 54 + 1 ) (but keeps it zero), and it can only decrease (or leave unchanged) 
the value of V - e~ Ui+1 ^ i+1+Si+1 ~ 5i + 2 ). Hence by ((3T|). the value of H(p) is only decreased. 

At the same time, j(u,5') > j(u,S). Note that in modifying 5, (|28|) need not hold for B and /. 
However, clearly there are other sets, B' and V with which it holds. For example, 

B' = {i£[l,J]:5' l <0}, /' = {»e[l,J]:<5<>0}, O' = 0. (41) 
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Moreover, 1 B', since we had 1 G" B and 6% was not modified. By the induction assumption we 
therefore obtain that H (p) > 0. 

Induction base. We show that for all (7, 5, z/, 5) satisfying (J2HJ), (EH), ©, such that 6) D I = 
and j(v,5) = J — 1, one has 77 (p) > 0. We have —vj^ij3j-\ + <5j_i — 5j > 0. Similar to before, 
we set <5j > (5j so that — vj-\l3j-\ + — 5j = 0. We claim that P{v,5') = 0. Indeed, since 
P(u,S) = {J - 1}, clearly P{v,5') C {J}. However, 

-uj(3j + 5j = -z^j/3j - vj^J-i + 
<0, 

where we have used the fact that J — 1 G P(u, 5) implies J — 1 7, and hence by ((2*5|) < 0. 
This shows that P(v, 5') = 0. As in the previous paragraph, because of the modification of 5, (|28|) 
need not hold for the 7 and B we started with, but there are other sets I' and B' (defined e.g. as 
in (|41|0 with which it holds. The results of Step 1 therefore apply, and therefore 77 (p) > 0. 

This completes the argument by induction and establishes (jllj) for superdifferentials. 

Verifying the PDE for subdifferentials. We are required to show that ()12j) holds for p G D~V(x), 
where x G G\d c G (and hence B(x) = 0). Unless D~V(x) is empty, the general form of p G D~V(x), 
x G G is 

where k £ A, and 

& < 0, i G I, Si = 0, i 7. (42) 

Here we have used the fact that V is the minimum of smooth functions, and therefore away from 
the boundary D~V{x) can have at most one element, equal to the gradient of any minimizing 
function. It suffices to show that 77 (p) < whenever p • 7i = —li=k(3k + <5« — <5j + i > for all i G 7. 
Using (ED, (jH and (JT3J), we find 

H{p) = c + A(l 
= c + A(l 

< c + A(l 

< c + A(l 
= 0. 



e ^- a i) + ^0V/Xi(l-e~ li=fc 



=fc/3/c+<5i-<5i+ 



i=l 



e A ) + ^/ii(l - e ~ ll = feA ) 
e^+^l-e^*) 
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